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Introduction 



We all know that we live in the. computer era. At the same time 
an eminent science writer has called our culture the psychological 
society (Gross, 1978). Inevitably, the two have met. At the 1983 
American Psychological Society meeting many vendors offered 
demonstrations of how various psychological functions could be 
evaluated on a computer. The Department of Defense is in the process 
of moving to computer controlled testing in recruit evaluation, one of 
the largest personnel selection programs in the world (Green et al., 
1982). On the other hand, misgivings about the rush to computerize 
have been voiced. An example of such a misgiving is Matarazzo's ( 1 9 83>. 
concern that the introduction of computers could change both the 
nature of testing and the interpretation of test scores. His remarks 
serve as a useful reminder that personnel evaluation is at its heart 
an exercise in psychology, not computer science. Applying computer 
technology is a means, not an end. 

In this paper we shall discuss some of the potentials and 
problems involved in computerized testing. There are two issues 
involved. One deals with the advantages or disadvantages of using a 
computer to administer -a test", regardless of the test's content. 
Most of the literature in the field has addressed this issue. The 
advantage seems to lie with the computer, for reasons that we describe 
immediately tslow. (We shall express a few misgivings). A far more 
difficult issue is whether or not the content of mental tests can be, 
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or should be, changed in computerized testing. The second issue will 
occupy most of our discussion. 

Tests of mental ability can be classified as individually 
administered or group administered tests. Individually administered 
tests are generally regarded as more valid, for two reasons. The 
examiner, functioning as a stimulus presentation device, can produce a 
variety of different auditory, visual, and even tactile stimuli. The 
same examiner, functioning as a control device, oan determine what 
questions are most likely to be informative about a particular 
examinee,. The paper and pencil form of gr>oup administered tests has 
none of these advantages but it is much cheaper. As a result, fixed 
format, paper and pencil testing is ox ten the method of choice for 
personnel screening involving large groups of people. Notable 
examples are tests used for personnel classification in education, the 
military, and the civil service. Paper and pencil tests have been 
criticized for their inflexibility and limited scope, but their 
cost-effectiveness has not been seriously challenged. 

Computer-controlled test administration (1) falls between the 
individual and the group testing situations. Almost any question that 
can be presented in paper and pencil format can be presented using a 
computer controlled, television-like display. Furthermore, it is 
relatively easy to program rules to make the next display contingent 
upon the response to past displays; i.e. to have the computer assume 
the control functions of the examiner in the individual testing 
situation. Finally, test scoring and administration are greatly 
simplified if responses are recorded directly by a computing system as 
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not others. The evidence is nixed in both' oases. Furthermore, it is 
not at all clear that the change produced by computerization would 
necessarily be a tiange for the worse. 

Responding to computerized tests evidently does require a 
specialized ability to deal with the test format. This was shown by 
Lansman et al. (1982) in a study that examined the correlations 
between computerized and paper and pencil tests of the "crystal iz'ed ■ 
and "visual" factors of intelligence that are specified in the 
Cattell-Horn theory of intelligence (Cattell, 1971; Horn and 
Donaldson, 1980). Lansman et al. found that the error components of 
the various computerized tests were correlated with each other, but 
not with the error components of the paper and pencil tests. This 
means that there are separate factors for item presentation mode, but 
that the presentation factors do not interact with other factors. 
Therefore tests presented in either format would be equally valid 
tests of intelligence factors, providing that the relative loadings of 
tests on the "psychological" and "mode" factors do not change across 
testing conditions. While definitive studies have not been done, 
there is evidence that this is the case. Computerized and paper and 
pencil versions of the Haven Matrix test appear to be essentially 
equivalent (Calvert and Waterfall, 1982; Watts et al . , 1982). This 
important because the Raven Matrix test is widely considered a good 
measure of general intelligence. 

The studies just cited involved homogeneous groups of subjects. 
It is possible that certain groups of subjects would be selectively 
favored or disfavored by computerized testing. The issue of 
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motivation is central. ' Some observers have claimed, more or less 
without data, that computer controlled tests can £e presented in a 
game-like format, and thus be self motivating.. Such an effect could t 
be specific to particular groups of individuals. Casual observation 
suggests that some people are indeed highly motivated by computer game 
formats. Others, find man-computer interaction intimidating, boring, 
or both. Intuitively computer games are very much associated with the 
youth culture (Loftuji and Loftus, 1983). Older people may view them 
with distaste. This is especially true if the testing procedure 
overtly m*" res speed of performance, for there is considerable 
evidence that older people dialike situations in which rapid 
responding is forced upon them (Hunt and Hertzog, 1982). To further 
complicate the issue, there is evidence that points the other way. 
European studies have found that computerized tests are more f~\ 
acceptable to elderly and handicapped people than are pen and pencil 

tests (Carr et al., 1982; Watts, Baddeley , and Williams, 1982). Given 

ft 

the small data baae, it would be inapproprate to conclude anything at 
all, yet, about the acceptability of computer-controlled testing in 
different populations. 

The iasue of motivation is closely related to the issue of prior 
experience. It is simply not true that "everyone" has or will have 
had extensive experience with« computer terminals during the 1980a. A 
more realistic assessment is that different people will have very 
different experiences. Playing arcade games that are computer 
controlled is not like using a text editor to write term papers. 
Intuitively, experience with keyboards Aand terminals would be expected 
to affect performance in a testing. situation. Marshall-Mies et al. 
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(1983) found a correlation of .2 (in a sample of 500 subjects) between 
self-report of experience with computer games and performance on a 
variety of computer-administered tests. Is this so small as to be 
trivial? The examinees were electrical power system operators who had 
an average of over eight years experience with computer controlled 
displays. Clearly the issue is not closed! 



In any case, it is well known that virtually any task that 
involves speeded reactions will show very large learning effects. 
This raises two concerns^ In assessing the f comparability of scores 
from different groups, a frequent issue in psychometri cs , care must be 
taken to ensure comparability of experience with the test apparatus* 
Within a single groupy the presence of large learning effects raises 
the question of stability in factor patterns. If a battery of 
computerized tests is gi*en repeatedly to the same people the mean 
values of the test scores will almost certainly change. Will the test 
covariances change? Will the correlations between test scores and , 
outside criteria change? These issues are relevant for any testing 
program, but may be particularly important for any computerized 
testing that involves the measurement of reaction time. 

One. of the strongest arguments for computerized testing isthat 
ip makes possible adaptive adjustment of test difficulty to the 
examinee's talents. It is conceivable that such a testing procedure 
could interact with personality traits. Consider an individual who is 
of slightly below average talent. Adaptive testing would begin by 
presenting this person with items of average difficulty, which will 
prove too hard. The examinee will thus be given an initial series of 
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failure experiences as the computer attempts to find an appropriate 
level of item difficulty. Now suppose that the individual is also 
test anxious, a well identified personality tra' (Sarason, 1980). 
Will the failure experiences early in the test cause the person to 
under perform on the remaining items? We simply do not know. 

There are two issues concerning "testing in the abstract" that 
revolve around the use of test results more than around the 
interaction between the examinee and the "W administrator. Tests of 
cognitive ability are typically used for one of two purposes; 
personnel classification or diagnostic evaluation. In personnel 
classification the purpose of testing is to identify i^ividuala who 
have (or lack), some degree of specific ability, so that these 
individuals may^be offered or denied an^assi,gnment . College entranoe 
examinatipns and Armed Services screening examinations are examples of 
classification tests. Classification testing is usually oriented 
toward the examination of broadly defined ability domains, such as 
verbal, spatial or reasoning skills. A relatively brief amount of 
time -is devoted to gaining in-depth information about a -person's 
ability"within one~of these domains"." Th"e>VTs"ex tensive predictive 
validity data showing that there is a correlation between test scores 
obtained in classification testing and similar broad criterion 
variables, such as academic achievement or success in service 
technical training schools (Burns, 1975; Jensen, 1980) . In assessing 
the validity of a classification method the appropriate criterion is a 
population one. Does the testing procedure Justify itself by 
providing data that reduce the costs of misclassi f ioation below the 
cost of the test fin n pnr" 1 ""™ biala? Computerized tests have to 
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show that they can meet this criterion, in competition with other 
methods of testing. Put(another way, suppose that the correlation 
between a oomputer administered test and a paper and pencil test was 
1.0. Then one should immedi xtely decide to use the method of testing 
with the lowest administrative ooats. This will seldom be the 

computer administered test. 

i/ 

The purpose of diagnostic testing is to gain a detailed knowledse 
of the pattern of performance of a specific individual. Absolute 
levels may be more important than levels relative to population norms. 
Neuropsychological examination 'is a prototype oase; the purpose of the 
examination is not to select a person for entrance into a fixed 
environment but to design an environment for the individual. Other, 
less extreme examples can be envisaged. The scores to be obtained and 
the criterion variables to be utilized will vary from case to oase. 
The introduction of computers into diagnostic testing hus may pose a 
much more difficult problem in validation than is the case for 
classification testing. On the other hand, the flexibility of 
computer administrated testing should provide a greater potential in 
the diagnostic than in the classification are*. 

Matarazzo (1983) raised a somewhat different issue. He was 
concerned that the widespread availability of computer-administered 
tests of intelligence and personality might lead to misapplication of 
the results. Matarazzo focused upon misinterpretation of and 
over-reliance on tests by naive users, simply because the tests were 
computerized. These issues, while real, are essentially issues that 
concern psychology as a professional guide rather than issues that are 



ERIC 



14 



Intelligence Testing Page 12 



directly relevant to psychology as a science. Concerns about oversell 
because something is on a computer are not qualitatively different 
from concerns about over-acceptance due to some other irrelevancy, 
such as endorsement by a famous athlete or entertainer. 



In discussing the scientific potentials for using computers to 
test intelligence, we must first define what it is that we are trying 
to test. Initially we shall take the Boring (1923) view that 
intelligence is defined by the intelligence test. Accordingly, we 
should first consider how computers can be used to expand upon current 
testing of those psychological functions that are tested by present / 
day methods. Such expansions are steps forward. We then take a leap 
into new areas, bu suggesting some psychological functions thatt are 
logically part of mental competence, but are not tested by present day 
methods. In discussing both steps and leaps we have attempted to 
distinguish between questions of test development, given the present 
state of knowledge, and questions that require further psychological 
research . 



HfldiXiflAUflJ] and Ezl&mlflfl of Present Annftnsnent Procedures 

Although there are many taxonomies of human mental performance 
there is also considerable agreement about the major intellectual 
dimensions that must be assessed in order to evaluate mental 
competence. Three types of -intelligence- will be considered; verbal 
ability, spatial-visual ability, and logical reasoning. While varying 
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studies have found these abilities to be moderately correlated in most 
populations, they clearly represent distinot aspects of human thought 
(Carroll. 1982). We will also consider the evaluation of memory, as 
memory assessment is part of individually administered test batteries. 

Verbal A illllJE 

Modifying onnnent. ass essment. "Verbal ability" refers to 
competence in comprehending linguistic messages. In order to exercise 
verbal ability on needs skills that go well beyond knowledge of word 
meanings. Nevertheless, on a statistical basis, people's ability to 
comprehend stories, utilize grammatical rules, and do other linguistic 
tasks are all well predicted by performance on a vooabulary test. 
(Carroll. 1979). Put another way, if we want to know what verbal 
ability is, one has to consider mmany things besides word knowledge. 
If all that is desired Is an estimate of a persons's verbal facility, 
a vocabulary test will often suffice (Carroll, 1971, Hunt, in press). 

One of, the open questions in intelligence testing is whether 
people differ in the size or in the nature of their vocabularies. The 
opinions of experts differ dramatically. Nickerson (2) has stated 
that people differ in their vocabularies by a factor of as much as two 
to one. On the other hand, Fillmore (1979) has stated that people do 
not differ very much in their total vocabulary size, but to differ in 
the sort of vocabularies they have. Neither Nickerson or Fillmore 
provide citations for their claims. (Carroll, 1971) cited data 
indicating that there are large individual differences on common 
vocabulary tests but thisdoes not settle the issue. In the 
traditional intelligence test a fairly.small set of words is 
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designated as "representative" of word knowledge in general, and the 
examinee is tested on this set of words. Such a procedure implicitly 
accepts the idea that ar ;t of a person's vocabulary is drawn from a 
universe of standard words. If testing were done by a computer it 
would be possible to probe knowledge of specialized vocabularies, by 
drawing test items from a population of words chosen to reflect the 
examinee's presumed characteristics. To take a simple example, if an 
examinee claimed to be a native speaker of Spanish it would be 
possible to probe for word knowledge about cars, if the examinee 
claimed expertise as an automCbile repairman, or for word knowledge 
about music, if the examinee claimed to a modern music fan. This 
procedure will be called tailored vocabulary testing. 

Tailored vocabulary testing can separate the Issue of the size 
and nature of a person's vocabulary. This could be important. 
Acquisition of a vocabulary depends upon two things: culturally 
determined exposure to a particular lexicon and personal sensitivity 
to verbal stimuli in general. Logically these are separate t*aits. 
Present tests, by measuring standard (American) English vocabulary, 
are undeniably biased toward measuring cultural exposure. For this 
reason it can be claimed that the tests are "unfair". If so, why do 
vocabulary tests serve as useful predictors? it could be that, in 
spite of their cultural bias, the tests still depend enough upon 
individual verbal facility to be useful measures of the underlying 
psychological trait. On the other hand, it could be that in some 
circumstances the tests predict future behavior precisely b ause they 
measure cultural background (3). Arguments could be made for either 
conclusion. A testing approach that evaluated a person's use of both 
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standard and specialized vocabularies would make it possible to 
investigate the issue further. 

».„ mentions. The co-ordination of verbal and non-verbal 
input can be looked upon as a prototypical verbal action. Tasks that 
require the action will be called -co-ordination tasks-. An example 
1. the comparison of verbal descriptions to directly perceivable 
visual, auditory, or tactile displays. Following Instructions often 
retires co-ordination; e.g. -Turn the knob t the left if the red 
light goes on.- There are substsntisl individual differences both in 
the strategies thst people use to co-ordinate verbal and non-verbal 
input, and in the effectiveness with which t^hey execute those 
strategies (Hunt, I960; in press). It appears that the sort of 

-verbsl ability- tested by oo-ordinstion tssks is partially 
..independent of -verbal ability- as tested by tasks requiring only that 

people respond to isolated words (Hunt, Davidson^, and Lansman, 1981), 

Most of the experimental paradigms used to study the 
oo-ordinstlcn of verbal and non-verbal information rely on computer 
presentation. Physicslly, it would be essy to adapt them for 
peyohometrlo testing. Two examples sre the ni.nt.lin.. TDrlf lugtl.n gnu 
1nl tn l r ..- r., 1n „,n. paradigms. In sentence verification the _ 
examinee must decide whether or not s verbal ststement correctly 
escribe, s Picture. The complexity of the verbal ststement correctly 
describe, a picture. The complexity of the verbal statement can be 
systematically manipulated (Carpenter and Just, 1975). Testing the 
ability to follow instructions, the examinee is told to execute 
actions contingent upon present.tion of certsin displays. Again the 
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variable of int.r.et 1. the examinee -a response- to ohar.es In the 
linguistic oo.ple.lty of the instructions, rather than to changes In 
stimulus or response complexity itself (Dixon, 19<"0. The instruction 
following paradigm is of interest heoause it tests the comprehension , 
of verbally stated procedures. Most present day verbal tests are of 
the comprehension of expository material. 



h 

There are a number of practical paychometrlc Issues to be 
resolved before either of these paradigms could be introduced into 
testing situations. They revolve around test-ret.st reliability, 
across time period, and across likely variations in testing 
situations. Such Question, fall somewhere between .basic" research 
.»<, -developmental studies-, but they must be answered before 
whether or not it is practical to test peoples ability to do 
„o-ordln»tlon tasks. Similar points will arise frequently in our 



6 

subsequent discussion 



Several computer-controlled procedures have been developed for 
use in the study of verbal comprehension process in general. Some of 
these involve highly intrusive measurement devices, such as 
instrument, for recording eye BO ve»a^s,-T*«^a» probably not _ . 
practical te.ting devices. A number of others laboratory paradigms 
involve what is called the ninnnn.rv UlU methodology. The mental 
effort r.culred to execute a linguistic task (e.g. parse a passive 
sentence) 1. measured by the ease of execution of an ancillary simple 
t..,, such as phoneme monitoring. Individual difference, can be 
evaluated using the secondary task paradigm, but doing so requires an 
elaborate experimental design that 1. probably not practical in most 
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testing situations (Hunt and Lanaman, 1982); 



Rapid single visual presentation (RSVP) is a teohnique for 
reading in whioh readers comprehend a message that is presented 
visually, one word at a time! (Potter, Krull and Harris, 1980). The 
rate of presentation can be controlled either by the reader or the 



oxaminer. To our knowledge 



texture material. It might 
computerized testing, but f 



the RSVP teohnique has not been used to 
evaluate individual differelnoes in the dynamioa of comprehension such 
as changes in the rate of reading as a function, of te complexity of 

be that the method could be adapted for 
great deal of work remains to be done to 
establish appropriate testing procedures. The RSVP teohnique seems 
particularly appropriate i[» situations in whioh the diagnosis of 
reading style is an issue. 

nonolualont Computer controlled testing is not likely to extend 
the range of evaluation ofj verbal ability in olassif ioation 



situations; if only beoauie the simple vooabulary test is suoh a good 
predictor. Therefore any Justification for computerizing verbal 
classification procedures will have to rest on oost effectiveness. 
Computer controlled testing may prove muoh more useful in diagnosing 
individual comprehension/ skills . Suoh testing may be quite valuable 
if there is reason to believe that the examinees have been exposed to 
some condition that would affect the non-lexical aspects of verbal 
processing. Por example, a variety of reports have indicated that. age 
deleterioualy aff eots/ the attention demanding, speeded aspects of 

& / 

verbal processing, although vooaulary size probably increases with age 
(Hunt and Hertzog, /1981 ) . Specific reading disabilities in 
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schoolchildren involve selective loss of j*e ability to deal with 
particular aspects of written material (filing 1979). Even more ^ 
dramatic selective losses of function arej sometimes seen in cases of 
brain injury. Prototype oomputer-oontroljled testing procedures for 
various aspects of reading comprehension jhave been developed 
(Frederiksen, 1 9 82) and cou^d be extended to more general situations 
where large scale, rapid diagnosis of language deficiencies were 
required. 



visual-Spa .^* 1 Ability 



' H M1fY ^ r ,,rr^ i.„. aam «nt. The term -spatial ability- refers 
to the ability to construct, examine, and manipulate internal 
representations of visual display*. Examples of standard spatial 
ability tests are the Minnesota Paper Ffrm Board test, the Primary 
Mental Abilities 'spatial, test, and tj. KohsBlock Design test (McGee 
1979). The well known "mental rotation" paradigms developed by 
Shepard and his collegaues (Shepard and Cooper, 1983) have also __ 
inspired some acre specific tests of spatial ability (Lansman et al. , 
1982; Vandenberg and Kuae, 1978). The domain of abilities covered by 
spatial tests is reasonably well fit by a two factor model (Egan, 
1979). One factor, -spatial encoding", is an ability to recognize a 
specified shape when it occurs in the visual field. Examples cf 
Encoding tasks are recognizing letters in a word or recognizing the 
triages embedded in a six pointed Star of David. The manipulation 
of a met** image is a rather different ability. It is best 
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illustrated by the rotation tasks developed by Shepard and his 
associates. A more prosaic example is the mental rotation of letters 
when reading a text that is presented upside down. We will refer to 
the aility to manipulate images as "visualization", although this is 

perhaps not the best term. 

Computerized testing methods offer a considerable potential for 
expanding the evaluation of spatial ability than in the verbal domain. 
Most preent spatial-visual tests mix the encoding and visualization 
factors. Pellegrino and his associates (Mumaw and Pellegrino, in 
press; Pellegrino and Kail, 1982) have shown that it is possible to 
obtain pure measures of both the visualization and encoding factors 
within the framework of a single test. This is done by presenting 
successive complications upon a single base problem (e.g. a form board 
completion item) in ways that increase the difficulty of the base 
problem along either the encoding or visualization dimension. Jhe 
item construction procedures developed by Pellegrino et al. could 
easily be combined with the response analysis procedures based on 
multicomponent latent trait theory ( Whitely , 1 98 1 ) and upon adaptive 
testing. This is an example of how the computer technology offers a 
chance to make a solid step forward in our evaluation methods. 

Hew Directions 

Our present ideas about spatial ability are based upon data that 
oan be gaterhed using the fixed format, paper and pencil testing 
technique. The paper and pencil format is much more restrictive in 
the spatial than the verbal domain, because the mapping between the 
dynamic, three dimensional, colored world and the two dimensional, 
static, (largely) black and white printed diagram is not as tifht as 
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the mapping between listening and reading. Hence there is a real 
potential for expanding our concept of spatial ability by adopting a 
new testing format. In particular, computer controlled displays can 
be dynamic. This is an important distinction, because spatial 
abilities arft thought to be involved in tasks in *hich motion is an 
inherent part of the environment, e.g. operating a vehicle or an 
aircraft. It would be relatively easy to test the ability to perceive 
and extrapolate motion directly by using computer controlled animated 
displays. Is the ability to deal with motion really predictable from 
our present static tests of spatial ability? If not, would measring 
it add to our ability to predict performance in tasks that we beleive 

require spatial ability? 

Traditional spatial ability tests deal with the ability, to look 
at a display. What about the ability to orient oneself to the 
surroundings? We will refer to this as -geographic orientation." 
Although there is evidence that people vary widely in their ability to 
maintain geographic orientattion {Thorndyke and Qoldin, 1981 )■ there is 
little systematic knowledge of the dimensions of the ability. The 
paucity of data may be due to the sheer economics of testing/Studies 
of geographic orientation require that people move about in the 
environment. Is it possible to conduct geographic orientation studies 
less expensively, 1>y usi-ng computer-controlled technology? People 
could be provided with the visual input from imaginary walks or 
drives, by combining computer controlled testing with te se of a 
random access video disk device. They could then be tested on their 
knowledge of the geography that they had experienced vicariously. 
Would the same people who did well in reconstructing the simulated 
geography do well at maintaining their orientation if they were moved 
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through a real environment? The experiments 
question are again straightforward. The answer itself s/important 
because of its implication for future research. If orientation nrtHi 
simulated environment correlates with orientation in the actual 
environment, then performance in the simulated environment ould be 
used either as a predictor of actual geographic orienttio* ability, or 
as a means of studying the relation betwenn .orientation and 
conventional measures of spatial ability. 

The above examples have concentrated oi| the "spatial" part of 
spatial-visual ability. Gopher (3) has pointed out that many (though 
not all) of the classical visual illusions fan be tested using 
computer generated displays. Several of these illusions are thought to 
indicate basic properties of higher order pierceptual^ systems. To 
illustrate, Coren and Girgus have shown that there are individual 
differences in susceptibility to various perceptual illusions, such as 
the Mulier-Lyer illusion. The dimensionality cf the space of 
susceptibility is considerably less than the number of illusions. 
Some of the factors of susceptibility seem to be related to peripheral 
visual processes while others seem to be assoclted with higher order, 
cognitive factors. Very little is know 6n how susceptibility *to 
illusions relates to other dimensions of/ intellectual or personality 
variation. (A striking exception to th^s statement is the very large 
literature on field dependency and rod^nd-f rame illusion (Witkin et 
\ 1954). If computer generated illusions can be produced reliably 

this could be a fruitful field of research. (4). 

' d^SJiW i ««»»t «' °«P uter •°" troll,d testing 5houl<1 

oonai'derably .xp.nd our abilit, to exa.ine soatlal-vlsua! abilities. 
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Enough research has already been done to warrant beginning the 
costructlon and validation of computer-oriented tests. Studies 
intended to expand our conceptualization of spatial ability to -nclude 
dynamic visualizations and geographic orientation could have an 
extremely high pay off. The extension of testing into the realm of 
perceptual illusions is more venturesome, but is worth considering. 



Reasoning 

HojUXyl Jtf Current As^H^nt. Reasoning is traditionally divided into 
two areas; deductive and inductive reasoning. Deductive reasoning 
requires that a general principle be applied to a particular case, 
whereas inductive reasoning involves extraction of a general rule or 
principle from examination of cases. Tests of reasoning have been 
widely used in the study of intelligence. Since the ti-ie of the 
ancient Greeks, deductive reasoning has been seen as the hallmark of 
rational thought (Johnson-Laird, 1983). Inductive reasoning 
(including analogical reasoning) has been shown to be among the best 
markers for the general ("g") factor in modern intelligence tests 
(Sternberg, 1977). This is intuitively reasonable, for a good 
argument can be made that the essence of intelligence is being able to 
transfer problem solving techniques learned in one situation to other, 
analagoua situations. 

During the 1970s and early 1980s considerable progress was made 
in understanding the psychological basis of both inductive and 
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deductive reasoning." Ther'e are now several, rather similar models of " 
how people represent various types of reasoning problems to 
themselves, and of how thosr representations are manipulated during 
problem solving. (Goldman and PeMegrino, 1984; Johnson-Lair^ and 
Steedman, 1978; Rips, 1983, Sternberg, 1977,1980,1982). These models 
provide good, explanations of why particular reasoning problems are 
easy or hard, and they identify microprocesses that must be exeouted 
in order to solve an inductive or deductive problem. The models could 
be used to design reasoning problems that vary systematically along 
known psychological dimensions, in much the same manner as was 
suggested for spatial tests.. Obviously, oomputer testing techniques 
could be used 4n such, an effort, but the effort itself does not 
require oomputers (see e.g. ,' Holzman, Pellegrino and Glaser, 1983). 
The cause for this conclusion is worth examining. Reasoning is, by 
definition, the task of operating upon an internal, conceptual 
representation of a problem. The oomputer provides a facility for , 
displaying different problem formats, but this should only affect the 
process of constructing the conceptual representation, not the process 
of manipulating it. Reasoning is also a prooess that often ta,kes place 
over minutes rather 'than milliseconds, and our interest typioally 
centers ^n errors as much as speed of reaction. While reaction time 
measures can be useful in/teat.ing Models that identify components of 
solution during reasoning tasks, (Sternberg's, 1977) it is unclear 
whether or not reaction time measures of component prooesses of 
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reasoning are sufficiently reliable to be useful in brief tests of a 
person's reasoning power (Goldman and Pellegrino, 198H) Thus the 
contribution of coaputerization to olssif roation teat* of reasoning is 
problematical. 

Computerized testing may have mbVe of a place when testing is 

done for diagnostic purposes. Reasoning problems are typically 0 

/ 

attacked in stages; encoding the elements of a problem, applying trial 
solutions, and evaluating solutions. Different strategies stress one 
or the other of these stages. Furthermore, suooess in problem solving 
appears to be related to the ohoice of a strategy. Sternberg and his 
associates haVe consistently found that good problem solvers spend 
relatively more time reading a problem and understanding the elements 
involved, whereas poorer probxem solvers seem to begin trying out 
solutions (Sternberg, 1982).' Dillon and Steve.nson-Hlcks (1981) have 
reported differences in problem evaluation. When' gtfod problem, solvers 
take multiple ohoioe reasoning tests, they examine the problem, 
develop a solution, and then look for the' solution in the set of 
answers. Poorer problem solvers spend more\time examining and 
rejeoting the alternatives provided in the answer set. Note that the 
former strategy generalizes beyond the multiple ohoice test format, 
while the latter strategy does not. Finally, Egan and Grimes Farrow 
(1982) have shown that peoples who adopt an abstraot public 
representation are more suooessful in solving transitivity problems 
than are peopl^e who adopt a ooncrete representation. A variety of 
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interactive computer controlled teating procedures could be used to 

< 

identify a, person's, style of problem solving, and perhaps to arrange 
ways of improving on it. But is this a good idea? The research cited 
aho** that different problem solving styles exist. St does not show 
how stable a person's problem solving .style is over different, 
occasions, or different situations. The problem of situational 
generality is of particular interest. 

maw mictions. Prom the viewpoint of a logician, the rules for 
inductive and deductive problem solving apply regardless of the 
content area. Psychologically content is far from neutral. Sternberg 
()1980) has pointed out that there are two stages of problem solving; 
-knowing abstract problem . solving methods and knowing when to apply 
them. The ability to realize that a probl > olving method is 
appropriate may be much rarer than the ability to apply a method in a 
familiar area. For example, mental retardates can be trained to apply 
problem solving procedures, but the application is tightly tied to the 
context in which it is learned (Campicne, Brown, and Ferrara, 1982). 
Cross cultural examples are also plentiful. Indeed, it seems that the 
i|idea that there are abstract reasoning procedures, apart from a 
particular context, is a peculiarly Western idea, that is acquired 
through contact with formal Western schooling (Soribner, 1977). 

Sternberg (1981) has recently advocated testing people's ability 
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to reason, in the abstract, by testing tne|* ability to deal with 
problems presented in what are almost bizarre* settings. An example is 
a problem about people who grow young instead of old. He has referred 
to the ability to deal with such situations as the ability to reason 
in "non-entrenched" situations. One can envisage a 
computer-controlled testing situation in which a perac-'s reasoning 
ability was examined both in settings with- whioh the examinee was 
familiar and inmore foreign settings. A related procedure would be 
to examine people's ability to use hints to suggest fruitful analogies 
in problem solving. This technique could be used to provide a finer 
analysis between individuals than the discrimination between -solvers' 
and f non solvers. 1 

Virtually every discussion of reasoning closes with a rather ill 
defined section on •problem solving.' Computer programs can be 
written to present quite challenging problem solving tasks-e.g. the 
ubiquitous 'dungeons and dragons' games advertised extensively in 
'computer Journals. Could these games be used to test problem solving 
skills? Although the idea is appealing, the idea does not always 
,ork. Time to solve a realistic computer-presented problem in 
electric power distribution correlated only -.16 (1-3*41) with 
supervisors' ratings of power system operators (Marshall-Mies, 1983). 

This discouraging result can be used as an illustration of a 
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pro..ic but important point. ThV concern, about teat length end item 
rellebUity tb.t ere routinely de.lt with in conventional t. .tlx ere 
eleo^ccnc.rn. in oo.put.r-ad.lnl.t.red teating. If • t.-t conaiete of 
only . few it..., aa in M.rahall-Miea et al.'a ... of a aingle 
proble., th.n It ie unV.aaon.ble to expect th.t the teat will be 
reli.hle. On the other hand, if there ere »eny it..., it la poaaibl. 
t0 produce e very 'reliable teat even though each item hea a 
. ub .t.nti.rit..-.p..ifio co.pon.nt (Green, 198,). Since r.aaonlng 
.»d problem aolvlng teak, are fluently found to have a oon.id.r.bl. 

a.ount of it.m-ap.clfio variance, extended beating i ntial if on. 

want, to evaluate a peraon-. general problem aolvlng or rea.onlng 
.blllty. To drive thi. point home, conaider a atudy by Sternberg and 
Gardner (1983). that reliably evaluated people', ability to execute 
particular component, of a r.aaoning atrategy. Such an evaluation 
. could be regarded a. tb. ultimate goal of di.gno.tio teating. 
Sternberg and Gardner-e participate aolved over on. tboua.nd problema 
each. Thi. llluatr.tea how .erloua the problem of item reliability 

I 

is. . 

r^i ,„,,»„■ C.n oomputer-oontroll.d teating help by ap.edlng up 
Item pr.aent.tlon and r.epona. evaluation! Speeding up thea. 
function, of teat ad.lni.tration would be only marginally u.eful. Ib. 
limiting f.ctor on .pe.d of r.e.onlng la al.o.t alw.y. the ap.ed with 
Iwhicb people thinx, not the ap.ed with which the probl.m 1. pr.aent.d 
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to them. Since the bottleneck is mental processing speed, speeding up 
test administration is almost irrelevant If there is to be a 
synergism between computerized testing and tbe evaluation of 
reasoning, it will be based on a sophisticated method for diagnosing 
problem solving 'strategies. 



Memory 



Memory evaluation has always been part of individually 
administered psychological tests. The Weohsler Adult Intelligence 
Scale (WAIS) contains a memory subsoale and a memory quotient score in 
addition to the more familiar intelligence quotient. (Matarazzo, 1972) 
In spite of their ubiquitousness, the memory soales used in 
psychometric tests are, simplistic compared to the measures of memory 
that are routinely utilized by experimental psychologists and 
neuropsychologists. The WAIS soale itself is inadequate to 
differentiate among types of amnesia, even though the instrument is 
frequently used to diagnose the amnesio syndrome (Hirst, 1982). 

The problem seems to be that the psylchometrio measures are, for 
the most part, based on a naive viev of h^man memory. Conventional 
memory subsoalea focus on the global functioning of two memory 
systems, very long term ("permanent") memory for facta* and memory 
span, the capacity to recall, exactly, material presented within the 
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last minute or ao. A more modern view of memory requires teating of 
at leaat three ayatema; the two deaoribed above and a third, working 
memory, that oontaina a general repreaentation of the otirrent 
aituation without neceaaarily containing an exact record of all 
detaila of the information that haa been preacnted (Baddeley, 1976; 
Hunt, 1977; Klapp, Marahburn, and Leater. 1983). Furthermore, moat 
experimental paychologiata would be unaatiafied with aimple meaaurea 
of capacity. In addition they would like to have a meaaure of the 
apeed of retrieval from varioua memory ayatema and aome meaaure of the 
rate of tranafer of information between ayatema. 

Procedurea for eatimating auch parametera are routinely uaed in 
experimental paychology today. Many of the procedurea demand the aort 
of control that can be achieved only by computerized teating. 
Furthermore, at leaat aome of the meaaurea are related to memory 
functioning in extra-laboratory aituationa. (Sunderland, Harris, and 
Baddeley, 1983). Thus there is clearly a potential for expanding 
present methods of aaaeasing memory in a way that would not be 
feaaible uaing conventional teating methoda. However, more ia 
required than aimply adding experimental paradigma into a teating 
battery. The paradigma generally have been deaigned to eatimate group 
parametera.. Whether or not they are aufficiently reliable to be uaed 
on an individual basis remaina to be ahown. Furthermore, moat of the 
experimental paradigma demand much more time from an individual 
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subject than would be available in a classification testing 
situations. Finally, a person's apparent memory capability may depend 
heavily upon a person's familiarity with the content of the 
information being memorized. It appears tht people will develop 
content-specialized frameworks for dealing with informatibn that they 
encounter repetitively. To take an extreme example, Chase and 
Eriksson (1981) report the case., of an individual who had a memory span 
for numbers in excess of eighty (i), because he had developed a 
special technique for memorizing inumbers. His memory span for other 
material was in the normal range. 

Results such as Chase and Eriksson's pose a serious r stion 
about the evaluation of memory. Is memorization ability a stable 
characteristic of the individual, or is it a product of the 
interaction between the individual's capabilities and his or her 
specific experiences? Theorists are .clearly divided on this point. 
Consider such a basic point as the development of short term memory. 
Some have argued that the development of an ever increasing short term 
memory capability is the cornerstone of .cognitive development. Others 
have argued that the apparent growth in memory is due to growth in the 
child's knowledge of memory strategies and due to a widened 
familiarity with useful, content-specific codes. (Siegler and 
Richards, 1982). The controversy is important in its own right. It 
is important to the development of computerized testing, because the 



9 

ERJC 



33 



Intelligence Testing 



Page 31 



testing procedure should be based on a clearer idea of what is being 
evaluated. 

Poking the issue as a contest between theories of 'generalized 
memory ability' and 'situation specific memory' is too simplistic. It 
is clear that if clinically normal individuals are given special 
training, so' that they develop 'expertise' within a content area, then 
their ability to remember information in that content area will be 
selectively improved. However, it is possible to develop tests of 
unfamiliar expository or arbitrary material. Most of the paradigms 
developed in the experimental laboratory are of this type. A 
substantial study of a performance on such tasks in a ; college student 
population (Underwood, Boruch, and Malmi, 1978) found that a very 
large percentage of the variance on all tasks could be accounted for 
by a general associative memory factor. This finding suggests that 
one or two memory tests would suffice to estimate general memory 
performance in a, variety of situations. 

The problem is thornier "hen 'exceptional' memory is to be 
assessed. The issues involved in evaluating exceptionally good or 
exceptionally bad memory are quite different. 

" J 
Extremely poor memory is usually associated with some 

pathological condition, such as Alzheimer's disease, senility, or 
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brain injury. Testing the affected individuals is necessarily a form 
of clinical interview. Computer-oontroiled presentation provides a 
chance to standardize the interview, which has some advantages, but 
the testing prooedure will probably always be an adjunot to individual 
patient assessment by a clinician. The role of computer controlled 
testing in the assessment of superior memory is rather different. As 
haa already been pointed out, oomputer controlled testing is flexible, 
so that within a fixed time it would be possible to test a given 
individual's memory for more different oontent areas, than could be 
evaluated using conventional testing. The remarks made earlier about 
tailored vocabulary testing are applicable. Thus computerized testing 
would be useful in identifying the fact that a person had superior 
memory for certain types of materials. 

What would be hard to do is to establish why an unusually good 
memory oapaoity existed. Superior memory oapaoity is generally 
associated with the use of ' elaborative strategies that are appropriate 
to the, material being memorized. (See Bransford et al., 1982 for a 
g ood discussion of this point and for a list of further citations.) 
There are a few situations in whioh strategies oan be inferred from 
oomputer-analyzable responses, such as reaction times (MacLeod, Hunt, 
«r.1 Mathews, ,1978) or eye movements (Dillon and Stevenson-Hioks , 
1981). in general, though, the best way to evaluate a person's memory 
strategy is to ask the examinee to describe it, and then have an 
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experimenter analyze a co»plex verbal response. While advances in 
computer comprehension of freely produced speech have been made, it 
seems unlikely that computers will be prepared to replace the 
experimenter in the near future. 



New Dimensions of Testing 



Learning, attention, and psychomotor skills are certainly aspects 
of mental competence. However, they are not usually evaluated in 
intelligence testing. Does computerized testing offer a chance to 
expand evaluation into these fields? 



Luannlng 

Definitions of what -the intelligent person- can do almost always 
include learning (Sternberg et al., 1981). Intelligence tests make 
relatively little effort to assess learning directly, because of the 
difficulty of doing so using a conventional test format. Since many 
of the paradigm* Led in laboratory studies of learning now utilize 
oomputer-oontrolledWrimental procedures, the advent of cheaper 
computing offer* the technological possiblity of assessing an 
individual's rate of learning. For example, it would be possible to 
observe a person's learning rate in situations as different as 
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learning to recite lists presented via computer or learning to play a 
video arcade game. 

Experimental studies of the feasibility of such testing have 
already been reported as part of an attempt to develop new military 
classification procedures. (5) Such proposals presuppose that there 
is such a learning ability apart from the ability to learn within 
specific content areas. The evidence for this proposition is far from 
clear. The assesment of learning ability raises a number of issues 
that transcend the narrower issue of computerized testing. It is 
instructive to see how the larger issues and the specific ones 
interact, 

V- TYP TT- ft* ^ning. Estes ( 1982), in a perceptive review of the 
issues, has pointed out that learning situations can be ordered by 
complexity. At the 'simple end' are studies of habituation and 
conditioning. Instrumental learning (operant conditioning) and 
paired-associates paradigms, are more complicated, while concept 
identification and verbal learning paradigms can be very complex. At 
the extreme, one could regard any education as a form of learning. 
Here the distinction between learning and induction beoomes arbitrary, 



The simpler learning paradigms, such as habituation and 
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conditioning paradigms, attempt to isolate the formation of elementary 
stimulus-response bonds. As Estes (Ld others) have pointed out, 
studies of extreme contrasts as the contrast between mildly mentally 



retarded and normal individuals have 



not shown particularly striking 



relations between performance in simple learning situations and 
measures of general intellectual competence. Thus such paradigms will 
not be considered further. 

jfne problem of evaluating a person's ability to learn 

complex material is complicated by three factors. When new facts are 
learned they are fitted into a person's existing information. , 
Therefore any evaluation of a person's ability to learn material with 
meaningful content will interact with the examinee's prior knowledge. 
A determination that a person is a rapid learner of material in 
history may have little predictive value for estimating the person's 
ability to learn auto mechanics. Since learning is highly dependent 
on memory, the remarks about individual differences in memorization 
strategies apply. Because individual differences in knowledge and 
atrategiea of information acquisition interfot, a person's learning 
ability may best be evaluated by constructing a model of how the 
learner conceptualizes the learning problem. There have been attempts 
to develop computer programs that do this, by observing students' 
responses to computer controlled presentations of material to be 
learned (Browned Burton, 1978; Collins, 1975; Sleeman, 1983). 
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Considerable research needs to be done before these programs are 
sufficiently validated so that their results oould be used in 
personnel selection or diagnosis. In fact, the time required to 
induoe a model of the learner may rule out the use of suoh student 
evaluation devioes in olassif ioation situations. The practicality of 
using learning modeling programs for a diagnosis of the learner's 
knowledge has not yet been established, but is a potentially 
interesting researoh ' topic The above remarks apply to studies of 
"learning ability" as evaluated within a single testing session. An 
alternative method of testing is to embed systematic assessment within 
oomputer assisted instruction programs, so that evaluation and 
learning prooeed hand in hand. If this oan be done the opportunities 
for testing inorease enormously. This approach also has the advantage 
of tying assessment direotly to the learning of material that 
presumably has value in its own right. Those instructional programs 
that attempt to develop models of the learner (see above) oould be 
regarded as sophisticated teaohlng and assessment situations. 



Procedural Learning. These somewhat disoouraging oonolusions are 
based largely on studies of the learning of verbal or quasi-verbal 
material. Learning also takes plaoe in situations in whloh the task 



extra-laboratory analog is to maohinery operation, surv ^Ll^noe of 
radar soreens, or even to athletio performance. Studies of experts in 



is to classify and respond to peroeptual displays. Here the 
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auch fields indicate that a great deal ofth.lr performance i. carried 

out in what haa come to be known aa the -automated .ode- of 
performing. R.aotlona to displaya are overlearned and immediate. 
Tnla ia oontraated to the .lower and attention demanding -controlled" 
mode of r.aponding. The .omewhat intuitive diatlnotlon between 
automated and controlled r.aponding ha. been operationally uefined by 
Schneider and Shiffrln (1977), in a aerlea of experiment, on the 
development of automated responding to vl.ual, dlaplaya. They found 
that automated re.pondlng could take literHlly thouaanda of trial, to 
develop, but that when It did develop reponae. might be aa much aa an 
order of magnitude fa.ter than reponae. made in the controlled mode. 
Subsequent .tudie. (Fi.k and^Schn.ider, 1983; Poltrock, Lan.man and •. ^ 
Hunt, 1982) have ahown that the automated-controlled distinction 
applies to a variety of item recognition situations, Quit, outside the 
narrow area of vlaual display recognition. Since the technique.' for 
evaluating automated and controlled reponding require very rapid 
presentations of stimuli and recording of responses, some form of 
computer control is necessary. Could the experimental procedure, be 
adopted to' testing situations? 

There are two basic questions that one would aak in atudylng 
individual differences in automated and controlled re.pondlng. Given 
a number of candidate, for training on a taak, which candidate, are 
likely to develop automated responding early in training? Given a 
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number of people who have already been trained to execute the task, 
which persons have developed to the stage of automatio responding? 
The first question is more interesting from the viewpoint of personnel 
selection, but unfortunately it may be the more difficult to answer. 
The straightforward way to ask the question would be to present 
candidates with a (computer controlled) simulation of the ta.sk to be 
learned.. or perhaps with the task itself.. and observe changes in 
performance over a relatively small number of learning trials. If 
learning rates early in training predict learning rates throughout 
training the approach should work. But do they? Here the evidence is 
mixed. 

e 

Schneider (Note 6) has presented data showing that learning 
performance early in training on a task analogous to interpreting a 
radar display is not predictive either of asymptotic performance or of 
the time required to learn an automated skill. Schneider studied a 
simple visual scanning task, analagous to the interpretation of a 
display on a radar screen. A somewhat similar design was used by 
Dunlap, Biladeau, and Jones (in press), but in their case the tasks 
were commercial video games that appear to involve visual scanning, 
target identification, and motor reactions. The factor structure of 
the tasks changed very little over extended praotioe. 

■ 

It is relatively easy to determine whether or not an 
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i„divldu.l h.a b... able to auto.at. performance on e epeoiflo tee*. 
Auto .ation la d.fin.d by inactivity or performance to incr.aaea in 
lnf or.atlon proo.aaing load.. Computerized evaluation procedure, 
could b. developed to monitor the etag. of a peraon-e training in a 
variety of eituatlone in which automation «a. Important. The 
evaluation procedure. .would be ap.oifio to tb. tea* at band, ratb.r 
tban b.ing general indloea of individual trainability. In general, 
U. .utomated-oontrol di.tlnotlon aeem. to be more relevant to 

^ o f-^nina crogram, than for predicting the 
evaluating the success of a training program. 

Success of a given . individual entering that program. 

" | 1 Tnnn r — * The above reBirka SUggeSt 

.trongly that learning in aituation .P*ciflo. Therefor, it ..... , 
..... to build «.nt into ap.oifio oompu,er controlled t.aohlng 

pr o gr .ma, but it ma.ea 1 »» " *'» *« ""^ 

proo .dur.a for evaluating learning ability in general. There are 
th o.a who would dl.a.nt with thi. vie-point. The dia.ent.ra believe 

. ..n.r.l learning factor that applies to more cognitive 
that there ia a general learnms 

ait uationa, euoh .. learning how to aolve probl... in mathematio. or 
reading. A central idea in their view ia the concept of a p.raon a 
-proximal .one of development.- Loo.ely, P.opl. «ho h.v. wide 
proximal xon.a are people who are able to utili,. bint, to learn how 
t0 .civ. unf..ili.r probl.... Th. concept wa. originally introduced 
by vygotaxy Ungliah tr.n.lation 1978). in the ,930.. and .PP.— to 
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h.v. influenced Soviet p.r.onnel evaluation., The Idea, although not 
the language, la si.ll.r to St.rnberg'a (1981) notion that on. ahould 
dl.tlngui.h between a p.r.on-. ability to do familiar (entrenched) 
ta.lca and hla or her ability to deal with novel, non-entrenched 
problem aolvlng. Outside of the Soviet union, Vygotsky's lde. h.a 
been tranalated Into action In a number of programs In which 
children-, potential, are a.....ed by giving them progre.slv.ly more 
hlnta until they can solve te.t problem.. The Idea 1. th.t the child 
i. simult.n.ou.ly being trained to be a problem .olver and bolng 
evaluated for the ability to deal with novelty. The word "child- 1. 
u..d .dvja.dly here, ae .11 the program, th.t we are aware of are 
directed toward children, »nd often abnormal children at that 
(Campione •«'.!., 1982! Feur.teln, 1980). At present the t.sk of 
evaluating . person-, potential for le.rnlng 1. conceived of ...» 
almo.t cllnlc.l interview, involving intense .ool.l Interaction 
between the examiner and the peraon being evaluated (tbla Is 
esp.cl.lly the case In Feur.teln. s work.) Recently, though, some of 
the testing procedures have been adopted for computer administration 

(Campione, Note 7). 

The apparent contradiction between the work of Feursteln and 

Campione et .1., on the one hand, and the other studies cited may be 

due to the population tested. It could well be that there are 

signlflc.nt indlvldu.l differences in gen.r.lized L.rnlng .blllty 

..ongst .low l..rn.rs, while le.rnlng might depend more on specific 
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prior knowledge in normal and rapid learners. This important issue is 
not directly related to the issue of the use of computers to assess 
learning ability. The answer to the question^ has strong implications 
for the design of computer programs intended to assess that ability. 



fny n hop° tQP skins 

intuitively, it makes sense to speak of people who are clever 
with their hands. On the other side of the coin, the evaluation of 
psychomotor skills is almost completely absent from current 
psychological testing. In part, this may be because it is hard to 
include a test of tool manipulation in a standard paper and pencil 
testing situation. One could imagine fairly simple psychomotor tests 
that could be administered by a computer. The simplest example would 
be some form of, tracking using an electronic -mouse", a light pen, or 
a similar pointing device. Other tests of tool use could no doubt be 
devised. Would there be any point in doing so? 

Psychomotor skills tend to be highly specialized and to be very 
sensitive to practice (Newell and Roaenbloom, 1981). Computer games 
that require psychomotor skills exhibit low session to session 
reliability until after ten or fifteen days of training (Kennedy, 
Carter, and Bittner, 1980). Finally, it has long been known that the 
relative importance of different components of a psychomotor skill may 



44 



Intelligence Testing 



Page 42 



O.ng. vitb tb. »t.t. or practice til-*-. 1967). «... finding. . 
,U .u««..t U.t tn.r. 1. nc g.n.r.i factor for p.ycbo.otor 
oc-ordination. If tbi. »■ then the fact that p.ycbo.otor t..a. 

... be .d.ini.t.r.d by oo.put.r 1. ao.ewbat irr.i.v.nt to fating 
„,.pt to ev.iu.t. the r..»lt. of .pacific training ..cu.noea. 

Th.r. i. a PO»ibl. .xc.pticn to thi. ....what di.oour.ging 
oon.iu.ion. it ... bo. oiai-c that' tb. .p.eo of a pernon-a 

eegnitiv. d.oi.ion proc can be r.v.ai.d by tn. .p..< of .i.pl. 

disor i.in.tion. and/or r.apon.. cn.ic. B,..pi.. - — »• < 
oh .io. r..ction ti.e paracign. in .niob a P.r.on .u.t »ov. on. of 

• sp ..d cf r.tri.v.i of infection for iong ».« 
r . qttlr . oo.put.r Pr...nt.ticn b.oeu.e of tb. - to octroi .ti.uiu. 

. *<.». «t Billlaooond acouraoy. 
pr.a.ntation and r.oord r.apon.. ti... at 

»..ld tn. .P..d of .i-Pl. cut rapid t..t. be includ.d in . 
.evening battery, Tn. .n.-.r to tni. Ration depend, upon two 

i..K4i4«>v and external validity. The 
familiar psychometric issues; stability and 

4.4-* Hntiea aoross sessions, is rather 
atability of choice reaction time duties, 

. *\ This contrasts with their extremely high 
low (approximately .6). This oontra 

< 9S) lntr . ion r.ii.biiity. Sunn . diacr.pancy indicate, that 

on cio. reaction ti urea be .object to intr.-.ubj.ot 

„. tb.t vary fro. day to day. Wbetb.r day to day variation. 
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mirror variation in other cognitive 
in a person's reaotion tines mirror v»"» 

b . h .vior 1. ai.ply not Wno-n. («. au.w.r to tM. ,...tlo. -ould be 
ln t.r..ting>. Oiv.n the f.ot of int.r-.ea.lon variation, it 1. 
,o...h.t aurpri.ing to find th.t th. oorr.l.tion b.tw.,. ohoie. 
r..otion ti.e. end .or. eo.pl.x t..k. 1. «» high a, it ia. 

Several .tudi.a hav. aho«n that ohoie. reaotion tl... are 

4. -f. «<• the "r" faotor of conventional 
reliably correlated with tests of the g 

.f<«* fHunt 1978: Jensen, 1982; Vernon, 1983). 
intelligence testing (Hunt, i*fo» 

f 2 to .H) are not sufficiently 
caution, though, that the correlations (.2 to .HJ 

.h.f thia test should be used as a Barker of general 
high to argue that this test onuuxu 

int.Uig.no.. «ueh 1... 1- of th. r.l.ti.n b.tw..n eo.pl.x 

b .„.vlor and p.ro.pti» dl.ori.in.tlon .p.ed. There is 
r .port of a very high oorr.l.tion between .ea.ur.d int.Uig.no. and 
,p..a of vl.u.l di.ori.in.tion (Brand, 1981 > , but aub..<u.nt wor* baa 
,au.d to r.plio.t. tb. origin.! etud, (Hettl.b.OK and *lrb,.19B3. 
Vernon, 1983). Tb. probl... of oontroUing for 3i.pl. a.n.ory 
defi.it., a»ob as a.tig..tia. or .yopia. b.v. app.r.ntl, not b..n 
studied. 

0 , ly a w.aa o... o.n b. .ad. for using oboio. reaotion ti.e and 
p. r0 .ptual di.ori.in.tion speed a. .e.sur.s of p.yohologle.l funo .on. 
U.t ar. .l.o ...sur.d by pr...nt d.y p.yoho.ot.lo t.ata. In p.rt 
Ul. i. « P"«« ..«•*«•- » °" 18 " ilUng t0 ,0 ° ,Pt ' 
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» tost a, a definition of general intelligence, why 

the Raven Matrix test as » "'^ 

or ha s only a borate correlation with the accepted teat, or both, 

t. *o regard information processing speed as 
more interesting approaoh is -o regara x 

Tf it oould be shown that 
.„ .xten.ion of »• *iUU.. te8ted - If 

.M — - . ...» — — « - ° g 

p . rfOrOi »0. in .on. olaa. or ooppX.X t..a tb.n » °< 
would be appropriate. 



I-' 

Attention 

Sp .. rB .n „m) .—.t- B.n.r.1 -tor 1. int.XXi g eno. 

„i ght in part be du. to p.rv..iv, umi— ^.renoe. in on... 

.U..t. .tt.ntion on th. problem at hand. S.varal 
•bility to oono.ntrat. »**■.»*»« 

th .r lit.r. .... aoboa. Spaar-.n-. o....nt. Por inat.noe. Mat.r.a, 
(19T2) ....rt.<, tbat in nc..X —a tb. obi.r ... - tb. ...or, .pan 
t..t on tb. ..ob.l.r Xnt.Hia.no. So.l. waa to t.at tb. .«.!.«•. 

* * m «Dite of such claims, early writers on 
ability to conoentrate. In spite 01 

lnt . mg .no. o„.r.< XittX. ..Pirioai evidence r.X.tin. 

U -attantion.. Tb. r.ilur. to d.v.Xop tb. id P-rtX, 

um .,», or .onitorin g a pereon-e attention.! .ffort. give, tb. 
0 o..tr.int. or conv.ntion.1 t..tin 8 .itu.tio... Xt waa aX.o p.rtX, 
*J to a raiXnr. to be p-.ei.a -bout wb.t ... ...nt b, -attention. 
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Sinoe 1970 the oonoept of attention has been artioulated more 
dearly. The most oommon view is to look upon attention as a 
resouroe, analagous/to eleotrioal power. Some authors argue for a 
single,, pervasive/ "power source" for thought (Kahneman, 1973), while 
others have presented models that postulate two or more sources of 
non-interohangeable resouroes (Navon and Gopher, 1979? Wickens, 1979), 
or competition for limited capacity information processing mechanisms 
(Hunt, 1981; Posner, 1978). In all oases, though, resouroe 
expenditure is measured by observing patterns of interference between 
two tasks. Consider two tasks, x and y, both of which demand 
attention. Examples are pursuit tracking and monitoring a stream of 
signals to detect the presence of a target. If the tasks oannot be 
exeouted together as well as each can be done alone, then the two 
tasks must compete for the same resouroes. The pattern of 
interference can provide a measure of the amount of attentlonal 
resouroes expended in such situations (Kerr, 1973; Norman and Bobrow, 
1975). 

Two paradigms have been used to study individual differences in 
the deployment of attention; both depend on some form of computer 
control for measuring reaction time and errors on a real-time basis. 
In the "secondary task" paradigm people are given two tasks singly, 
and then asked to perform them together. In "switohing" or 
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•filtering, p.r.dlg.. p.opl. »ust either switch fro. monitoring one 
source of input to .onltoring another (e.g. .niton fro. .onxtorlng 
elgnele in on. ..r to .onltoring slgn.l. presented to th. other) or 
they »ust .onltor on. slgn.l .ouro. while Ignoring another. Th. 
presence of a a.condary tank «ay Introduce a .ouro. of individual 
variation that 1. not apparent when p.opl. p.rfor. t..k. .lngly. Thie. 
could b. either beo.u.e th. inor.a.ed de.and. of th. e.oond.ry tssk 
provld. an aoour.t. .e..ur. of the total att.ntion.l resources 
available to a p.raon (Hunt and L.ne.an, 1982) or b.oaue. there are 
reliable individual dlff.r.no.. In th. ability to alloo.t. att.ntlon 
to oo.p.tlng task. (Stankov, 1983). 

A so.ewhat different view of attention stresses the control a 
person has over attention alloo.tion. rather than th. total a.ount of 
.tt.ntion.l resource, ev.il.ble. Ooph.r (1982) hs. developed this 
Idea extensively using s dichotio listening paradlg.. Th. .xa.ln.. 
listens to th. diohotlo.lly presented str.a.s of signals. On cue s/h. 
...t switch rapidly fro. att.ndlng to th. infor.atlon presented in on. 
.. r to attending to infor.atlon presentee in the other. Th. dependent 
variable is th. speed with which th. switch of att.ntlon can b. 
executed. Several studies have shown that th. att.ntlon switching 
p.r.dlg. can be used to dl.orl.ln.t. good and bad operators of a 
variety of .aohin.s, ranging fro. bus.s to airplanes. Th. 
correlation, are not large, and th. validity of pr.diction .ay d.p.nd 
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upon the examinee'a familiarity with the criterion taak. On the, other 
hand, the claaaif ication deciaions being made in theae aituationsVy 
involve aubatantial economic commitment. (This ia certainly the cas> 
in pilot training.) Therefore even amall Increaaea in predictability 
can be valuable. j 

Given the data now in hand, meaaurea of attentional reaourcea and 
control warrant aerioua conaideration for incluaion in aome 
claaaification fata, especially if the goal ia to predict performance 
in aituations where rapid decision making ia required. The way that a 
test should be used, though, will depend very much upon the particular 
aituation, becauae there are a number of theoretical and practical 
queations to be resolved. Pe -pa the most important of theae ia the 
iasue of situational generality of attention measurea. Conaider 
Gopher'a procedure. The paradigm involves auditory perception, but ia 
preaented as a teat of the ability to control attention in general, k 
recent atudy by Lanaman, Poltrock, and Hunt (1983) indirectly 
queationa thia aaaumption. College atudenta took a variety of 
auditory and viaual teata that required the control of attention. 
There was evidence for aeparate factors for control of auditory and 
viaual attention, rather than for a general -attentional control- 
factor. While Gopher's test was not among thoae uaed by Lanaman et 
al., their reaults do suggest that further atudiea ahould be done to 
determine the factorial complexity of 'attention' itself. 
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Attention i. said to 'w.x and wane', thus implying that It i. 
.object to day to day, and perhapa minute to minute variability. 
While variability might be of interest in itaelf, (See the remarks 
.bove concerning reaction ti»4). any test to test variability po.es 
.amlni.tr.tiv. problem., be/.u.e extended testing 1. required, benign 
problems are also po.ed be4.u.e the inv..t..nt of .tt.ntion is no more 
directly obs.rv.bl. th.n/i. the investment of intelligence. Attention 
1, evaluated by p.rfor/.no. on a task. Individu.l differences in 
.kill on the task ar/oonf ound.d with individu.l difference, on the 
.mount of .tt.ntion/d.vot.d to it. Thi. very much oomplio.tes the 
design, required to i.ol.t. m...»r.. of .tt.ntion from .... r.. of 
t..k .pecific .ill. (Ack.r».n, Wlcken., Schn.id.r, 1982; Hunt .nd 
L.n.«.n, 198 2 / De.ign i..u... r.th.r th.n technology, m.y prove the. 
limiting feature in effort, to .v.lu.te .ttention»l c.p.biliti... 
N.v.rtn.l.,.., giv.n the d.t. now in h.nd, we r. G .rd the ext.n.ion of 
testing to the ev.lu.tion of individu.l difference, in .tt.ntion .. 
one of the mo.t promising .pplio.tlon. of computer controlled tenting. 



/ 

/ 

t 

/ 

/ 



Validation ia one of the thorniest problems in assessment. What 
paychologists would like to do is to talk about the relation between 
test scores and "performance- in some extra-laboratory task. 
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Establishing extra-laboratory performance is usually a substantial 
problem. Very few aspects of everyday life are evaluated in anything 
like an objective manner. Perhaps the easiest situation to deal with 
is the prediction of academic performance, where grades are regularly 
assigned. Yet anyone only mildly familiar with academics is 
uncomfortable with grade point averages as an evaluation of student 
performance. The situation is even worse in the workplace, where 
competence must often be measured by such surrogates as economic 
reward or observer ratings, variables that are subject to influences 
other than personal ability. From a narrow point of view, the 
validation problem for computer presented tests is no different than 
the validation problem for any other test. The problem lies in the 
validation soore, not the test soore. 

Computers could be used to collect measures of human behavior in 
extra-laboratory situations. The most obvious, and perhaps the most 
controversial, use is to monitor individual work performance. This is 
feasible in industries-where computer readable records are already 
used for' management control of inventories, customer flow, or other 
resources. Examples are check-out counters in supermarkets, air 
traffic control centers, and airplane ticket sales. The recording 
devices used to keep track of goods or paperwork could be used to 
•atimate human productivity. Such an application raises serious 
social issues, which we shall not comment upon. We do point out that, 

i 
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seen strictly fro. the viewpoint of a psychologist interested in 
assess.ent f keeping records on individual workers in their normal work 
environment is an attractive solution to some of the problems that 
have plagued psychometrioians for years. Beoause the measures would 
be directly related to work, they would be face valid. There is a 
danger, of course, that measures that could be recorded would assume a 
-psychological validity- that they ought not have, to the expense of 
other, less easily recorded aspects of performance, but this argument 
could be made against the collection of any leas-than-perf eot 
performance measure. A second feature of continuous monitoring is 
that it records normal performance. Overt testing, on the other hand, 
is likely to yield measures of (near) maximal performance. The 
prediction of average performance may be much more important in the 
workplace, 'in the past, little work-has been done on this because 
somewhat paradoxically, average performance is more difficult to 
assess in an objective manner than is peak performance. 

In training situations many of the advantages of continuous 
assessment can be retained without incurring some of the social 
disadvantages. This is particularly true if the training is to be 
conducted using simulation techniques that are themselves under 
computer control, since it may be easier to include computer 
monitoring of performance in a simulated task than in an actual task. 
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The combination of aaaeaament with computer controlled simulation 

F 

may represent an important extenaion of teating to the workplace. 
Simulationa are becoming increasingly popular training devioes, 
•specially in situationa where training on the criterion taak itaelf 
is not feasible. Furthermore, simulation training sessions are leas 
artificial than teat situations. A teat must be relatively brief and 
must be meaningful for untrained people. A simulation taak may take 
hours, and it may be practical to train people for several days before 
they can operate the simulator. Simulator performance itaelf muat be 
evaluated againat actual performance. Once thia ia done, though, the 
simulator stands at an intermediate point between brief, inexpensive 
testing and the expenaive, error prone evaluation of extra-laboratory 
performance. It may be economically feaaible to screen candidate" 
teats by examining the correlation between teat and Simulator 
performance. Tests that paaa thia examination can then be evaluated 
by the more expenaive prooeaa of collecting data on test and 
extra-laboratory work performance. 



/ Concluding Remarka 



If the decision to move to computer controlled testing is to be 
•tde either on the grounds of eaae of adminiatration or greater 
accuracy of evaluation of what 'we now evaluate, then the deciaion 



54 



Intelligence Testing 



Page 52 



should be dictated by economic considerations. Wven that there exist 
two tests of the same psychological function.one computerized and one 
in a more traditional format, which is the most cost effective? 
Psychological issues are raised only when it is asserted that 
computerized tests are 'better' measures of cognition than are 
conventionally administered tests. This is a claim for a, 
psychological gain that is distinct from economic gain. 

Providing that the purpose of testing is for personnel screening, 
it does not appear -that there is very much psychological gain to be 
obtained from computer controlled testing of reasoning, verbal 
intelligence, or psychomotor abilities. The psychological functions 
that make up these domains are either not changed by the format of the 
test, or (in the case of psychomotor functioning) are highly specific 
to the particular task being used. Ve are somewhat more optimistic 
about the use of computer controlled testing if the purpose of an 
examination is diagnosis of individual performance. In this case much 
more extensive testing' is possible, and computerized administration 
may make it possible to tap aspects of a person's language processing 
or problem solving behavior that would not be reliably revealed over a 
few minutes testing, regardless of the format used. 

Computer controlled testing is likely to have much more of an 
i.paot on the study of individual differences in spatial-visual 
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reasoning, memory, and on the study of individual differences in 
attention. In these three fields there are a number of interesting 
test situations that are simply not feasible without the use of 
computer controlled testing stations. Research on the development and 
validation of suoh teat procedures oould begin today. 

Perhaps one of the most challenging questions for future research 
is whether or not learning potentials can be evaluated by having / 
people solve problems while being coached by. a computer program. At 
present, questions about learning potential are asked almost , 
exclusively about children, and the research presumes an interaction 
between the child and a dedicated, highly skilled instructor. It is 
conceivable, but by no means certain, that interactive computing 
techniques oould be used '.o identify learning potential in adults. It 
is unlikely that research in this area would result in procedures that 
would be useful for initial personnel screening, because the time 
required to assess learning potential is likely to be on the order of 
hours. It may be possible to assess adult learning potential in 
diagnostic situations, where the organization has a commitment to the 
individual. Given the anticipated reduced size ^ the workforce over 
the next fifty years, combined with the rapid introduction of new 
technology, diagnostic testing may become much more widespread than it 
is today. If so, high priority should be given to research on the 
assessment of adult learning potential. 
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Computers are fascinating, flexible, and highly useful tools. In 
presenting a somewhat mixed evaluation, we feel almost as if we are 
taking a stand against progress. Nevertheless, we feel that caution 
is in order. Computers do raise many possibilities for testing and 
training. Whether these possibilities are useful ones, though, is not 
a question that can be answered by demonstrating engineering 
feasability. The questions that have to be answered are 
psychological, and the answers can only be obtained by appropriate 
psychological research. 



Author Notes 



The preparation of this paper was supported by the Office of 
Naval Research, Contract N000 1H-80-C-063 1 to the University of 
Washington (Earl Hunt, Principal investigator) and by Contract 
N000H-81-C-0616 to the university of California, Santa Barbara (James 
Pellegrino, Principal Investigator,). The preparation of this report 
was both inspired and greatly facilitated by the discussion at a 
conference on the use of computers in intelligence testing, sponsored 
by the Office of Naval Research and held at x the University of 
California, Santa Barbara in March, 1983. We gratefully acknowledge 
the comments of the participants at that conference and of Dr. 



ERJC 5? 



Intelligence Testing 



Page 55 



Marshall Farr, of ONR Peraonnel a*d Training Program who had 
originally propoeed that theae issues should be explored in this and 
other forums. The report itself, however, is not in any way a summary 
of the proceedings of the conference, nor would its conclusions 
necessarily be endorsed by any conference participants other than 
ourselves. The opinions expressed here are our own, and do not 
represent the views of the Office of Naval Research or the U.S. 
Government . 



Footnotes 



1. One o.n Imagine some extremely exotic computer controlled 
tests. We restrict our dl.ou.elon to the aort of fating that would 
b. pc.albl. on computer ay.t.m that one could re.aonably expect to be 
available for very large t.atlng programs. It will be aaaumed that 
the terminal ocn.l.ta of a standard 512 x 512 dot, black and whit, 
cathode ray tube (CHI) monitor and that the response devloe 1. either 
. typewriter style keyboard, acme for. of panel operated by button 
presses or, when specifically mentioned, a light pen or similar analog 
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traoking devioe whose position oan be sensed by the oomputer. This 
definition oovers suoh devioes as the paddles and steering wheels 
associated with oomputer games, touoh sensitive soreens, and drawing 
devioes suoh as the electronic -mouse". We will consider systems with 
voice generated output, but not systems with speech recognition 
capability. Finally, no consideration will be given to devioes that 
can recede electrophysiological signals into computer readable form. 
While active research on <the relation between electrophysiological and 
behavioral measures on cognition is being conducted, we do 1 not believe 
that electrophysiologic! measures will be utilized in personnel 
evaluation in the near future. There will, no doubt, be important 
special exceptions to this statement. 



2. Nickerson, R. Comments on papers in the symposium on 
individual differences. American Educational Research Association 
Meetings, Montreal, March 1983. 

3. Gopher, D. Remarks on individual differences in attention. 
Presented at the Conference on Individual Differences. Univ. of CA, 
Santa Barbara. March 1983. 

U. Some illusions may be \ery difficult to produce using the 
aort of computer testing configuration that we envisage. Virtually 
all ftlimHTd display systems presenting computer testing or 
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00. municative use acne variety cf the standard 'television" display, 

1. e. 520 x 520 dots, refreshed at 60 Hz. Display changes fas^ enough 
to produce or vary an illusion »ay not be possible in all oases. 

i 

5. Payne, D. OSAF pr.s.ntatlon at the Trl-Servlo.e Cognitive 
Solenoee Synth.al. Conference, H.v.l Post Graduate School, Monterey, 
CA, March 1984. Kyllon.n, P., Pr.a.nt.tlon at Office of Naval,' H.aearoh 
Contractors' Meeting on Individual Difference* and Information 
Prooeaslng, fale university, Maroh, 1984. 

. ■ I 

I 

6. Schneider, .. Remark, on Individual difference. In automatic 
spending. Conference on Individual Difference.. Oniv, cf CA, Sent. 
Barbara, March, 1963. / 



/ 



7. campion., t. Child D.v.lcpment end Ment.l Ret.rd.tlon C.nt.r 
Colloquium. University of Waahington, February 1984. 
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Boiling Air Force Base 
Washington, DC 20332 

' 1 Dr. Earl A. Alluisi 
HQ, APHRL (AFSC) 
Brooks AFB, TX 78235 

l 

I 4r. Raymond E. Christal 

AFHRL/MOE " 

Brooks AFB, TX 73235 

1 Bryan Dallman 
AFHRL/LRT 

Lowry AFB, CO 80230 

x 1 Dr. Alfred R. Fregly 
AF0SR/NL 

Boiling AFB, DC 20332 

1 Dr. Genevieve Haddad 
Program Manager 
Life Sciences Directorate 
AFOSR 

Boiling AFB, Dp 20332 

I Dr. John Tangney ( 
AFOSR/ NL 

Boiling AFB, DC 20332 

I Or. Joseph Yasatuke ~ 
AFHRL/LRT 

Lowry AFB, CO 80230 
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12 Defense Technical tnforaatlon Center 
Cameron Station, Bldg 5 
Alexandria, VA 22314 
Attn: TC ' *<7H 7*4- 

^ I Jordan Graf man, Ph.D. <• 

•Department of Clinical Investigation 
Walter Reed Army Medical Center 
6825 Georgia Ave., N. W. 
Washington, X 20307 

,1 Military Assistant for Training and 
Personnel Technology 
Office of'tne Under Secretary of Defens 
for Research & Engineering 
Room 30129, The Pentagon 
Washington, DC 20301 

1 Major Jack Thorpe 
0ARPA 

1400 Wilson Slvd. 
Arlington, VA 22209 
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Civilian Agencies 

1 Or. Patricia A. Butler 
NIE-BRJN Bldg, Stop # 7 
1200 19th St., NW 
Washington, DC 20208 

I Dr. Paul G. Chapln 
Linguistics Program 
.National Science Foundation 
Washington, DC 20550 

l Dr. Susan Chipman 
Learning and Development 
National Institute of Education 
1200 19th Street NW 
Washington, DC 20208 

1 Edward Esty 
Department of Education, OERi 

MS 40 

1200 19th St., NW 
Washington, DC 20208 

I Edward J. Fuentes 
National Institute of Education 
1200 19th Street, N. W. 
Washington, DC 20208 



1 Gloria Gilmer 
National Institute of Education 
1200 19th St. N.W. 
Mail Stop 7 
Washington, DC 20204 

i rA&E , T*L 
National Institute of Education 
1200 19th St., NW 
Washington, DC 20208 

1 Or. Arthur Maimed 
724 Brown 

U. S. Dept. of Education 
Washington, DC 20208 

, * • 

1 Dr.' Andrew R. Molnar * 
Office of Scientific and Engineering 

Personnel and Education 
National Science Foundation 
Washington, DC 20550 

« Or. Ramsay W. Selden ' 
National Institute of Education 
1200 19th St., NW 
Washington, DC 20208 1 
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1 Chlaf , Psychological Reserch Branch 
0. S. Coaat Guard (G-*-l/2/TT42) 
Washington, X 20593 

** 1 Dr. Frank Wi throw 

U. S. Office of Education 
400 Maryland Ave. SW 
Washington, X 20202 

1 or. Joseph L. Young, Director 
Memory & Cognitive Processes 
National Science Foundation 
Washington, DC 20550 
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Private Sector 

1 Dr. John R. Anderson / 
Department of Psychology / 
Carnegie-rtellon University 
Pittsburgh, PA 15213 

1 Dr. Alan Baddeley 
Medical Research Council 
Applied Psychology Unit / 
15 Chaucer Road / 
Cambridge CB2 2EF / 
ENGLAND / 

1 Dr. Patricia Baggett / 
Department of Psychology 
University of Colorado 
Boulder, CO 80309 

1 Dr. Isaac Bejar 
Educational Testing Service 
Princeton, NJ 08450 

I Dr. John Black 
Yale University 
3ox 11A, Yale Station 
New Haven, CT 06520 

1 Dr. Glenn Bryan' 
6208 Poe Road 
3vithesda* MD 20817 

1 Dr. Jaime Carbonell 
Carnegie-Mellon University 
Department of Psychology. 
Pittsburgh, PA 15213 

I Dr. Pat Carpenter 
Department of Psychology 
Carnegie-Mellon University 
Pittsburgh, PA 15213 

I 

1 Dr. John B. Carroll 
409 Elliott Rd. 
Chapel HJjll, NC 27514 

1 Ps^Wllllaa Quae 
Department of Psychology 
Carnegie MellOfl' 0\llTCfSlr«y 
Pttrteburgh, PA 13213 

1 Dr. Michcline Chi 
Learning R&D Center 
University of Pittsburgh 
3939 O'Hara Street 
Pittsburgh, PA 15213 
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Private Sector 

t Or. William Clancey 
Department of Computer Science 
Stanford University 
Stanford, CA 94306 

I Or. Michael Cole 
University of California 

at; San Diego 
Laboratory of Comparative 
luman Cogni.ci.on - D003A 
La Toil*, :\ '52093 

I Dr. Ulan M. Collins 
Bolt Beranek & Newman, Inc. 
50 Mouiton .Street 
Cambridge, MA 02138 

I Dr. Lynn A. Cooper 
LRDC 

University of Pittsburgh 
3939 0'Hara Street 
Pittsburgh, PA 15213 

I SRIC Facility-Acquisitions 
4833 Rugby \venue 
•v- Bethasda, .40 20014 

I Mr. Wallace Feurzeig 
Opi::a3nt of Educational Technology 
lolc 33ranak 4 'te-.ni a*. 
10 Mouiton Sc. 
Cambridge, \i\ 02238 

I Or. Dexter Fletcher 
'll"AX Research ..Institute 
1875 S. St'ate St. 
Oreo, UT 22333 

I Dr. John R. Frederlksen 
Bolt Beranek & Newman 
50 Mouiton Street 
Cambridge, MA 02138 



I Dr. Don Gentner 
Canter for Hunan Information Processing 
University of California, San Diego, 
U folia, Z\ 01093 

I Dr. Dedre Centner 
Bolt Beranek i Newman 
10 Mouiton St. 
Cambridge, MA 02133 
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1 Or. Robert Glaser 
Learning Research & Development Center 
University of Pittsburgh 
3939 0'Hara Street 
PITTSBURGH, PA 15260 

1 Dr.. Marvin D. Glocte 
" 217 Stone Hall 
Cornell University 
Ithaca, NY 14853 €> 



1 Or. Joaph Goguen 
SRI -International 
* 333 Ravenswood Avenue 
•Menlo Park, CA 94025 

i Or. Daniel Gopher 
Faculty of Industrial Engineering 

& Management 
TECHNI0N 
Haifa 32000 
ISRAEL 

1 Dr. Bert Green- 
Johns Hopkins University 
Department of Psychology 
Charles & 34th Straat 
Baltimore, MO 21218 

DR. JAMES G. GREEN0> 
LRDC 

UNIVERSITY OF' PITTSBURGH 
3939 0'HARA STREET 
PITTSBURGH, PA 15213 



1 Or. Barbara H^yes-Roth 
Department of Computer 

. Stanford University . 
Stanford, CA 95305 
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1 Glenda Greenwald, Ed. 
Human Intelligence Newsletter 
P. 0. Box 1163 
Birmingham, MI 48012 

1 Dr. Lloyd Humphreys 
Department. of Psychology 
Universi of Illinois 
603 East Daniel Street 
Champaign, IL 61820 
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Private Sector 

I or. Marcel Just 
Department of Psychology 
Carnegie-Mellon University 
Pittsburgh, PA 15213 

I or. Scott Kelso 
Hasklns Laboratories, 1 Uc 
270 Crown Street 
New Haven, CT 06310 

I Dr. David Kleras 
Oepartaent of Psychology 
'Jaiversity of Arizona 
Tuscon, kZ 35721 

I or. Walter '<lntsch 
Department of Psychology 
University of Colorado 
Boulder, CO 80302 

1 Dv, Stephen Koaslyn 
1236 .niliaa James Hall 
33 Kirkland St. 
1 abridge, MA 02138 

I oc. Pat Langley 

Tha Robotics t i3tlCi: j 

' irnagla-Mellon UniverjUy 

Pittsburgh, PA 15213 

I Dr. Marcy Lansman 
The L. L. Thur9ton^ ?*7ihoaetrlc 

Laboratory 
University if Morth Carolina 
Davie Hall 013A 
Caapel Hill, NC 27514 

I Dr. Jill Larkln 
Department of Psychology 
Carnegie Mellon University 
Pittsburgh, PA 15713 

L Or. Alan Lesgold 
'.earning R&D Can».»r 
Iitvarslty of Pittsburgh 
3939 O'Hara Street 
Pittsburgh, PA 15260 

I Dr. Jim Levin 
University of California 

-it %*n Diego 
Laboratory fof Comparative 

Human Cognition - O003A 
La Jolla, CA 92093 
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Private Sector 

1 Dr. Michael Lev^na 
Department of Educational Psychology 
210 Education Bldg. 
University of Illinois 
Champaign, IL 61801 

I Dr. Charles Lewis 
iFaculteic toclila Wetenschappen 
Rijlcsuniv.arsL : ilt Croningen 
Oude Boteringestraat 23 
9712GG Groningen 
Netherlands 

1 Dr. Robert Linn 
College of Education 
University of Illinois 
Urbane, IL 61801 

1 Dr. Don Lyon 
AFHRL/OY (U0RI) 
Williams AFB, AZ 85225 

1 Dr. Jay McClelland 
Department of Psychology 

MIT 

Cambridge, MA 02139 

I Or. Allen Monro 
Behavioral Technology Labor itorles 
1845 Elena Ave., Fourth Floor 
Redondo Beach, CA 90277 

1 Dr. Donald A Morman 
Cognitive Science, C-015 
Univ. of California, San Diego 
La Jolla, CA 92093 

I Or. Jesse Orlanatcy 
Institute for Defense Analyses 
1801 M. Beauregard St. 
Alexandria, VA 22311 

I Dr. James A. Paulson 
Portland State Unlvjrslty 
P.O. Box 751 
Portland, OR 97207 

I Dr. James W. Pellegrlno 
University of California, 
Santa Barbara 
Dept. of Psychology 
Santa Barabara, CA 93106 
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Private Sector 

- I Dr. Nancy Pennington 
University of Chicago 
Toil nte School of Business 
liOl S. 58th Sc. 
Chicago, IL 60637 

I OR. PETER POLSON 
DEPT. OF PSYCHOLOGY 

UNIVERSITY OF COLORADO j 
'JOULDER, CO 30309 

'1 or. Mike Posner 

Department of Psychology 
University of Oregon 
Eugene, OR 97403 

I Dr. Fred Re if 
Physics Department 
Univarsity of California 
aerkelay, CA 94723 

I or. Lauren Ri3nick 
LRDC 

University of Pittsburgh 
>939 O'Hara Street 
Pittsburgh, Pa 1521 

I Dr. Vadraw M. Rosa 

American institutes f >»* Research 
1055 Thomas Jefferson St. NW 
Washington, DC 20007 

««»• I Dr. Ernst Z. Rothkopf 
Sell Laboratories 
Murray H1.1L, MJ 079 n 

I Dr. William B. Rouse 
Georgia Institute of TVchiwlo^y 
School of Industrial & Systems 

Engineering 
Atlanta, OA 30332 

I Dr. David Rumelhart 
Center. for Human Information Processing 
Uaiv. of California, San Diego 
La Joila, CA 92093 

1 Dr. Michael J. Samet 
Perceptronics , Inc 
6271 Variel Avenue 
Woodland Hills, CA 91364 
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\1 Dr. Walter Schneider 
\ Psychology Department 
\ 603 E. Daniel 
Champaign, IL 61820 

\ 

1 br. Alan Schoenfeld 
Mathematics and Education 
The University of Rochester 
Rochester, NY 14627 

\ 

1 Dr. Edward E. Smith 
Bolt Beranek & Newman, Inc. 
50 Moulton Street 
Cambridge, MA 02138 

* t • ■* 

1 Dr. Richard Snow 
School of Education 
Stanford University 
Stanfori, fl\ 94335 

1 Dr. Kathryn T. Spoehr 
Psychology Department 
Brown University 
Providence, RI 02912 

I Dr. Robert Sternberg 
Dept., of Psychology 
Yale University 
Box UA, Yale Station 
Mew Haven, CT 06520 

1 Dr. Kikumi Tat9uoka 
Computer 3ased Education Research Lab 
252 Engineering Research Laboratory 
Urbana, LL 61801 

I Dr. Maurice Tatsuoka 
221 TUcatjLon Bldg 
1310 1Uth St. 
Champaign,, IL 61820 

1 Dr. Perry W. Thorndyke 
Perceptronics, Inc. 
545 Middlefield Road, Suite 140 
Menlo Park, CA 94025 

1 Dr. Douglas Towns 
Univ. of So. California 
Behavioral Technology Labs 
1345 S. Elena Ave. 
Radondo Beach, CA 90277 
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I Or. Kurt Van Lehn 

Xero* PARC 

1333 Coyote Hlil tload 
Palo Alto, CA 94304 
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• I Or. 0*711 J. Weiss 
N660 Elliott Hall 
University of Minnesota 
75 E. River Road 
Minneapolis, HN 55455 

I Or. Keith T. Wescourt 
Perceptronlcs , Inc. 
545 MiddlefleU Road, Suite 140 
Menlo Park, CA 94025 

I William B. Whit! .n 
Sell Laboratories 
2D-610 

Holndel, NJ 07733 

' u 1 Or. Christopher Wlckens 

Department of Psychology 
University of Illinois 
Champaign, IL 61320 

1 Or. Thoma9 Wlckens 
Department of Psychology 
Franz Hall 

Uslvursity of California 
405 HlLgarde Avenue 
Los Angles, C\ 90024 

I Or. Mike WllLiaras 
lolt J2r1n.sk ^ *Jjtfan 
10 feuUon it. 
Cambridge, HA 94304 
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